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This report describes an attempt to revise and 
pilot-test an achievement cruiz, a background information form, and an 
attitude inventory considered inadequate when used at the Summer 
(1968) Institute on Evaluation at the University of Illinois. In the 
oriainal achievement quiz, most items tested recall or information 
and few tested the student’s ability to comprehend. In the revision, 
items were written at a transformation or evaluation level. Based on 
a field test at Lovola University, an item analysis of the data 
indicated an improvement. The rpvised background information form was 
constructed to te more selective among applicants and thereby to 
improve future institutes. Both a second and third edition of the 
attitude inventory were tested in a field sample of 29 Loyola 
students to determine the relationship of the inventory’s factors to 
the individual’s nerformance on the achievement quiz and his 
hackaround information form. The overall objective of the institute 
was to develoD a climate among educators supportive of evaluation. 
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STATEMENT of FOCUS 



This supplemental report describes an attempt to: 
revise and pilot-test three instruments* 
considered inadequate when used at the 
Summer Institute on Evaluation (1968) 
and 

clarify the program's objectives, data, continuity 
by constructing and superimposing an evaluation 
model or schema on CERLI's Report of the 1968 
Summer Institute on Eva luation, University of 
Illinois, July 29 - August 9, 1968 (October, 1968). 

Achievement Quiz, Background Information Form, 
Attitude Inventory 



August, 1969 



T 



I. REVISIOI! of INSTRUMENTS 

A. The Achievement Quiz 

Analysis of the original achievement test revealed that 
12 (among 31) items were within the acceptable difficulty 
and discrimination range. In the revised instrument, these 
"good" items were included among 50 items testing the 
following aspects of evaluation and distributed according 
to the following ratios: 



Experimental design - 


16% 


Data analysis 


- 4% 


Evaluation model 


14% 


Educational process 


- 4% 


Objectives 


8% 


Standards 


- 4% 


Statistics 


8% 


Imput data 


- 4% 


Variables 


8% 


Interviewing 


- 2% 


Data collection 


6% 


Norms 


- 2% 


Instrument development 


6% 


References 


- 2% 


Judgments 


6% 


Samp les 


- 2% 


Outcomes 


6% 







In the original quiz, most items tested recall or information 
and appreciably little of the student’s ability to comprehend. 

In the revised quiz, items were written at a transformation or 
evaluation level and for the most part dealt with the performance 
objectives cited in the original report: 

It was expected that each participant would — 

Design components of an evaluation plan for local use 

Make a table of contents for a summative evaluation 
report of his own local gifted program 

Improve his facility for using the language and 
concepts of measurement and evaluation 
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Prepare to try out certain standardized classroom 
observation techniques that con be useful in 
evaluation studies 

Examine prototyp evaluation reports and read 
selections from the literature on evaluation 

i 

Work out solutions to a series of problems 
, designed to simulate the conditions and 

circumstances of local gifted projects in 

Illinois 

In order to involve the student in a task simulating what 
an evaluator actually would be expected to experience 
and perform, the testwriter constructed items related to 
evaluating the objectives, outcomes, experimental design, 
instruments used, etc. of a proposal submitted by a par- 
ticipant in the Evaluation Institute at Urbana. (See Appen- 
dix A- Achievement Quiz.) 

M 

Two techniques were used to evaluate the revised version of 
the Achievement Quiz. Twenty-nine graduate students in 
educational psychology at Loyola University (Chicago) 
during the spring semester (1969) comprised the field test 
sample. Three evaluation experts (directly involved with 
evaluation at CERLI) took the quiz to discover items of 
disagreement. 

Based on the field test at Loyola, an item analysis of the data 
indicated that 14 items were within approximate acceptable 
* limits (difficulty .50; discrimination .30, plus range) and 13 
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items could easily be improved to come within acceptable 
limits. (See Table 2.) However, the criterion used here 
for the item analysis was the total test score which formed 
a high-scoring and low-scoring criterion group. Better 
criteria could be investigated. The remaining 33 poor items 
will be retained in the test until another sample can be 
found or until the post- test is administered to the field 
test sample. 

The reliability of the test based on the pre-test data col- 
lected at Loyola University was .31. This is a split half 
reliability coefficient that can be increased to a reli- 
ability of .47 for the whole test when based on the Spearman- 
Brown prophecy formula. Correlations were also calculated 
between the achievement quiz, the background information 
data sheet and the attitude inventory. 

If two out of the three experts accepted a particular 
statement, the item was retained in the revised achievement 
quiz. (See Table 1.) The only exceptions to this rule 
were items 22 and 23, where the view of the testwriter was 
considered to be the correct one. If there was a three- 
way disagreement on an item, the item was subject to later 
revision. Items 14, 25 and 28 needed to be revised. 

On 29 items, there was complete agreement among the three 
expert evaluators. On 17 items, there were two experts in 
agreement (although on two items the exceptional response 
was retained) and three items had a three-way disagreement 
and were revised. 
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TABLE 



n 

j 



RESULTS OF EVALUATION EXPERTS’ SCORING 



1. 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 

9. 

10 . 
11 . 
12. 

13. 

14. 

15. 

16 . 

17. 

18. 

19. 

20 . 
21 . 
22 . 

23. 

24. 

25. 

26. 

27. 

28. 

29. 

30. 

31. 

32. 
33* 

34. 

35. 

36. 

37. 

38. 

39. 

40. 

41. 

42. 

43. 



Expert 

I 



Expert 

II 



Expert 

III 



Summary 

Key 



2 

2 

4 

4 

4 

2 

1 

1 

2 

1 

1 

4 

4 

1 

2 

1 

1 

2 

1 

1 

2 

3 

1 

3 
1 

4 
3 
2 
2 

3 

4 
2 



2 

2 

2 

1 

4 

4 

2 

3 

4 
4 
3 



2 

2 

4 

4 

4 

2 

1 

1 

2 

1 

1 

4 

4 

4 

2 

1 

1 

4 

1 

2 

2 

4 

2 

3 

3 

4 

3 
1 
2 

4 

3 
2 
2 
2 
2 
1 

4 
4 
2 

3 

4 
2 
3 



4 

2 

3 

4 
4 
2 

4 

2 

1 

1 

4 

2 

2 

1 

1 

2 

1 

1 

2 

4 

2 

3 

4 

3 

2 

3 

4 
2 
2 
2 
2 
1 
4 
4 
2 

3 

4 
4 
3 



Im 

2 

4 

4 

4 

2 

1 

1 

2 

1 

1 

4 

4 

Revised , 1 
2 
1 
1 
2 
1 
1 
2 

Accept 3 
Accept 1 

3 

Revised , 1 

4 
3 

Revised, 2 
2 

3 

4 
2 
2 
2 
2 
1 
4 
4 
2 

3 

4 
4 
3 
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TABLE 1 (continued) 

RESULTS OF EVALUATION EXPERTS’ SCORING 









44. 

45. 

46. 

47. 

48. 

49. 

50. 



Expert 

I 



4 

4 

2 

2 

2 

4 

1 



Expert 

II 



4 

4 

2 

2 

3 

4 
1 



Expert 

III 

4 

4 

2 

3 

2 

1 



Summary 

Key 
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TABLE 2 



ITEM ANALYSIS OF THE ACHIEVEMENT QUIZ DATA 

(N = 24) 

Mean = 20.8 Standard Deviation = 4.2 



Discrimi- 

nation 



Difficulty 



Acceptable 

Limits 



Easily 

Improved 



1 . 


.24 


.52 


X 




2. 


.08 


.36 






3. 


.16 


.64 






4. 


. 16 


.24 




X 


5. 


.08 


.44 






6 . 


.00 


.24 






7. 


.00 


.24 






8. 


.24 


.44 


X 




9. 


.24 


.52 


X 




10. 


-.08 


.20 






11. 


. 16 


. 16 




X 


12. 


.32 


.80 




X 


13. 


.40 


.52 


X 




14. 


.40 


.44 


X 




15. 


.08 


.12 






16. 


.08 


.68 






17. 


.24 


.44 


X 




18. 


.08 


.52 






19. 


.48 


.64 


X 




20. 


-.08 


.04 






21. 


.24 


.44 


X 




22. 


.08 


.G': 






23. 


.32 


.48 


X 




24. 


.24 


.28 




X 


25, 


.00 


.08 






26. 


. 16 


.24 




X 


27. 


. 16 


.24 




X 


28. 


.00 


.08 






29. 


.32 


.16 




X 


30. 


-.08 


0 44 






31. 


.32 


.44 


X 




32. 


.08 


.20 






33. 


.16 


.72 




X 


34. 


.08 


.76 






35. 


.24 


.76 


X 




36 . 


* 16 


.56 




X 


37. 


.08 


.44 






38. 


-.08 


.36 






39. 


. 00 


.32 






40. 


.00 


.56 






41. 


.32 


. 56 


X 




42. 


.48 


.48 


X 
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TABLE 2 (continued) 






ITEM ANALYSIS OF THE ACHIEVEMENT QUIZ DATA 

(N = 24) 

Mean =20.8 Standard Deviation = 4.2 

Discrimi- Acceptable Easily 

nation Difficulty Limits Improved 



43. 


.16 


.48 




X 


44. 


.32 


.40 


X 




45. 


.00 


.56 






46. 


.00 


.72 






47. 


.16 


.32 




X 


48. 


.24 


.20 






49. 


.16 


. 56 




X 


50. 


.24 


.84 




X 
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B. Background Information Form 



The Background Information Form (See Appendix B) was designed 
to collect data useful in grouping participants in the pro- 
gram and to identify numerous variables that could be used 
in making predictions about participants in a similar 
evaluation institute. 

At the 1968 summer institute on evaluation (Urbana) , the 
existence of two somewhat incompatible groups of partic- 
ipants created problems in communication and professional 
and personal relationships. The fact that some participants 
had an extensive background in statistics and others lacked 
this experience could account for the dissonance. The fact 
that the total group included both administrators and class- 
room teachers also polarized the trainees’ interests and 
participation. 

The original application form for the summer workshop in 
evaluation provided only routine information and very brief 
responses to the following questions: 

Please describe, briefly, your previous research -braining 
and experience 

Please describe any evaluation efforts you are currently 
engaged in 

Will you be in a position next year to do some kind of 
evaluation of your district 's gifted program? 

The Background Information Form has been constructed to 

procure information about the participant’s educational back- 

. ground, professional development, and familiarity with 

evaluation terms and concepts. Thus, the form may achieve 

-9- 
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more selectivity among applicants and a more meaningful 
institute. 

The final question (#20) in the form lists 80 statistical, 
measurement and evaluation items about which the applicant 
indicates his degree of understanding. These responses 
. reflect the level of each student and thus educational ex- 
periences complementing the participants’ development and 
experiences can be planned. Even though asking students 
to rate their level of understanding is subject to much 
error, the data can be scored and subsequently correlated 
with other data as well as used in structuring the insti- 
tute . 

The graduate students who had "taken" the Achievement Quiz 
also comprised the sample for field-testing the Background 
Information Form. Of particular significance were their 
scores on item #20 that could subsequently be correlated 
with performance on the Achievement Quiz and the Attitude 
Inventory. Students who had completed one full course in 
Statistics and Tests and Measurements (indicated in their 
responses to items 8 and 10) were able to score higher in 
judging their level of familiarity and understanding of the 
terms and phrases. 

C. Attitude Inventory 

The second edition of Stake’s Attitude Inventory was 
administered to the 29 "field sample" Loyola students in 
order to determine if the total inventory score or the 



o 
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inventory’s factors (identified by the Center for Instruction, 
Research and Curriculum Evaluationf-CIRCE at Urbana) could be 
compared with their performance on the Achievement Quiz and the 
Background Information Form. 

Three of CERLI’s evaluation experts also were asked to complete 
the attitude inventory to provide a basis of comparison and 
indication of the goodness of the item. CIRCE supplied the 
scoring key. 

The items in the Attitude Inventory and comparative data 



showing the respective distribution of scores are presented 



in the following text: 



1. "The evaluator of an educational program should identify 
unanticipated outcomes as well as anticipated outcomes." 

Agree Disagree No Response 

Students 73% 17% 10% 

CERLI 100% 0% 0 

CIRCE Key X 

2. "It is important for the program evaluator to find out 
how we^.1 various people like the program." 

Agree Disagree No Response 

Students 86% 14% 0 

CERLI 100% 0 0 

CIRCE Key X 

3. "Generally speaking, a program should be evaluated with 
reference to one or more ’control’ programs." 



Agree 

76% 

67% 

X 



Disagree 

21 % 

33% 



No Response 
3% 

0 



Students 
CERLI 
CIRCE Key 



4. 



\V 



"The evaluator should accept the responsibility of finding 
the strongest, most defensible, and publicly attractive 
points of the program." 



Students 
CERLI 
CIRCE Key 



Agree Disagree 

55% 45% 

0 100 % 

X 



No Response 
0 
0 



5. "In evaluating a program, it is at least as important to 
study and report on the types of teaching as it is to 
study and report on the amount of learning." 





Agree 


Disagree 


No Response 


Students 


83% 


10% 


1% 


CERLI 


100% 


0 


0 


CIRCE Key 


X 






"The evaluator 


should draw a 


conclusion as to 


whether or 


not the goals 


of the program 


are worthwhile." 






Agree 


Disagree 


No Response 


Students 


69% 


28% 


3% 


CERLI 


100% 


0 


0 


CIRCE Key 


X 







7. "It is more important to evaluate a program in comparison 

to what other programs do than to evaluate it with reference 
to what its objectives say it should do." 



Students 
CERLI 
CIRCE Key 



Agree Disagree 

14% 86% 

0 100 % 

X 



No Response 
0 
0 



8 . 



"The proper attitude for the evaluator is one 



of skepticism." 



Students 
CERLI 
CIRCE Key 



Ag ree 

28% 

33% 



Disagre e 

69% 

67% 

X 



No Response 
3% 

0 



9. "The task of putting the objectives into writing is more the 
task of the evaluator than that of the educator." 



Students 
CERLI 
CIRCE Key 



Agree 

21 % 

100 % 

X 



Disagree 

72% 

0 



No Response 
7% 

0 
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"It is essential that the full array of educational objectives 
be stated before the program begins," 





Agree 


Disagree 


No Response 


Students 


79% 


21% 


o’ 


CERLI 


0 


100% 


0 


CIRCE Key 




X 




"As a rule 


, an educator should 


inform students 


as to the 


objectives 


of the training," 








Agree 


Disagree 


No Response 


Students 


90% 


7% 


3% ' 


CERLI 


100% 


0% 


0 


CIRCE Key 


X 







12. "The major purpose of educational evaluation is to describe 
what specifically is happening." 





Agree 


Disagree 


No Response 


Students 


48% 


52% 


0 


CERLI 


67% 


33% 


0 


CIRCE Key 




X 





13. "At the present time education has more of a technical and 
mechanistic orientation than it should have," 



Students 
CERLI 
CIRCE Key 



Agree Disagree 

66 % 21 % 

33% 67% 

X 



No Response 
13% 

0 



14. "The job of an evaluator is mostly one of finding out how 
well students learn what they are supposed to learn." 





Agree 


Disagree 


No Response 


Students 


69% 


28% 


3% 


CERLI 


33% 


67% 


0 


CIRCE Key 


X 






"An educator should be encouraged to revise 


his objectives 


throughout 


the training period." 








Agree 


Disagree 


No Response 


Students 


59% 


34% 


7% 


CERLI 


67% 


33% 


0 


CIRCE Key 


X 








-13 



"The process of decision-making is one of th£ itfeakest 
links in the present operation of the schools." 

Agree Disagree No Response 

Students 59% 21% 20% 

CERLI 100% 0 0 

CIRCE Key X 

"Educators have important aims that cannot be stated 
adequately by anyone in terms of student behaviors." 

Agree Disagree No Response 

Students 31% 59% 10% 

CERLI 100% 0 0 

CIRCE Key X 

"The entire school day and the entire school experience 
should be divided up and assigned to the pursuit of stated 
educational goals." 





Agree 


Disagree 


No Response 


Students 


41% 


52% 


7% 


CERLI 


100% 


0 


0 


CIRCE Key 


X 






"The first job in 


instruction is 


the formulation of a 


statement of objectives," 








Agree 


Disagree 


No Response 


Students 


79% 


21% 


0 


CERLI 


67% 


33% 


0 


CIRCE Key 


X 






"A teacher should 


tell his students any and 


all of his 


teaching objectives." 








Agree 


Disagree 


No Response 


Students 


38% 


59% 


3% 


CERLI 


67% 


33% 


0 


CIRCE Key 




X 




"The major purpose 


: of educational evaluation 


is to find out 


the worth of what 


is happening." 








Agree 


Disagree 


No Response 


Students 


76% 


17% 


7% 


CERLI 


100% 


0 


0 


CIRCE Key 


X 
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22. "The evaluator should be a facilitator more than a critic 
or reformer or scholar." 



Students 
CERLI 
CIRCE Key 



Agreed Disagree 
62% 38% 

100 % 0 

X 



No Response 
0 
0 



23. "To expose students to something and to arrange a particular 
experience for students are respectable educational goals, 
needing no specification as to the effects they will have 
on the students," 



Agree Disagree No Response 

Students 24% 72% 4% 

CERLI 33% 67% 0 

CIRCE Key X 

24. "Evaluation requires an understanding of what the conditions 
were before the treatment began." 

Agree Disagree No Response 

Students 100% 0 0 

CERLI 100% 0 0 

CIRCE Key X 

25. "An educator should be encouraged to conceive of his 
objectives in terms of how student behavior will be altered." 





Agree 


Disagree 


No Response 


Students 


83% 


14% 


3 


CERLI 


100% 


0 


0 


CIRCE Key 


X 






"The main purpose 


of evaluation 


is to gain understanding of 


how things work." 


Agree 


Disagree 


No Response 


Students 


41% 


48% 


11% 


CERLI 


33% 


67% 


0 


CIRCE Key 


X 







27. "Description and value judgment are equally important 
components of evaluation." 



Students 
CERLI 
CIRCE Key 



Agree 

69% 

100 % 

X 



Disagree 

28% 

0 



No Response 
3% 

0 
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28. "In conducting an evaluation, there is no justification for 

the exercise of subjective judgment of any kind by the evaluator," 



Students 
CERLI 
CIRCE Key 



Agree Disagree 

28% 66% 

0 100 % 

X 



No Response 
6 % 

0 % 



29. "An evaluation of an education program should include a 

critical analysis of the value of the goals of the program." 



Students 
CERLI 
CIRCE Key 



Agree 

90% 

100 % 

X 



Disag ree 

10 % 

0 % 



No Response 
0 % 

0 % 



30, "The strategy of evaluation should be chosen primarily in 
terms of the particular needs the sponsors have for 
evaluation data." 



Students 
CERLI 
CIRCE Key 



Agree Disagree 

55% 35% 

67% 33% 

X 



No Response 
10 % 

0 % 



31. "The educational evaluator should attempt to conceal all 
of his personal judgment of the worth of the program he is 
evaluating. " 



Students 
CERLI 
CIRCE Key 



Agree 

31% 

33% 



Disagree 

69% 

67% 

X 



No Response 
0 % 

0 % 



32. "The sponsor of an evaluation should have the final say-so 
in choosing or eliminating variables to be studied." 



Students 
CERLI 
CIRCE Key 



Agree D i s agree 

14% " 72% 

0 100 % 

X 



No Response 
14% 

0 % 



33. "An evaluator should concentrate on gathering data about 
the costs and benefits of the program." 



Students 
CERLI 
CIRCE Key 



Agree Disagree 

35% 59% 

33% 67% 

X 



No Response 
6 % 

0 % 
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34. 



It 



"Cognitive goals are more important in education than ariective 
goals , " 



Students 
CERLI 
CIRCE Key 



Agree Disagree 

28% 59% 

33% 67% 

X 



No Response 




35. "An evaluator finds it almost impossible to do his job 
without intruding upon the operation of the program at 
least a little." 

Agree Disagree No Response 

Students 66% 28% 6% 

CERLI 67% 33% 0% 

CIRCE Key X 



36. "All important educational aims can be expressed in terms 
of student behaviors." 



Agree 

Students 41% 

CERLI 0% 

CIRCE Key X 

37. "Some educational goals are best expressed in terms of 
teacher behaviors . " 



Disagree 

55% 

100 % 



No Response 



0 % 



Students 
CERLI 
CIRCE Key 



Agree 

59% 

67% 



Disagree 

35% 

33% 

X 



No Response 
6 % 

0 % 



38. "The principal purpose of evaluation is to test out the 
theories and hypotheses." 



Students 
CERLI 
CIRCE Key 



Agree 

62% 

0 % 



Disagree 

35% 

100 % 

X 



No Response 

3% 

0 % 



The Attitude Inventory administered to the Loyola students 



contained only two items similar enough to items in the inventory 



used with summer institute participants to warrant comparison, 
(Though many ideas were incorporated in the second edition, none 
of the original items in the first edition of the inventory was 
retained verbatim,) 
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For examp le. 



First Edition, I tern Number Two "The evaluator should det- 
ermin'^ Whether the ’"goals' of a program are worthwhile." 

Second Edition, Item Number Six: "The evaluator should draw 
a conclusion as to whether or not the goals of the 

program are worthwhile." 

Agree Disagree Don T t Know 

Urbana Participants - 

Pre-test 50% 38% 12% 

Field Test Participants 69% 28% 3% 

Key X 

First Edition. Item Number Seventeen: "It is up to the local 

educator to rule out the study of a variable because it 
is not one of his objectives." 



Second Edition. Item Number Thirtv-Two: "The sponsor of an 

evaluation should have the final say— so in choosing or 
eliminating variables to be studied,," 

Agree Disagree Don * t Know 



Urbana Participants - 

Pre-test 4% 77% 
Field Test Participants 14% 72% 
Key ■X 



19% 

14% 



In lieu of factor analyzing the second edition, a third edition 
of the inventory was developed and factor analyzed. (Some items 
in the second dition were deleted, some were rephrased, some 
items were added.) 

There was some disagreement between the CERLI experts and the 
CIRCE Key obtained from Urbana. The three CERLI experts agreed 
with the key on 21 items; two out of three experts agreed with 
the key on nine items; and on the remaining eight items, two- 
thirds of the experts disagreed with the key on seven of the 
eight items (12, 14, 20, 23, 26, 30 and 37); and all experts 
disagreed with the key on the remaining item (36). However, 
items 30, 36 and 37 appear on at least two of the scales in the 
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attitude scale of the CIRCE third edition attitude inventory with 
both the Agree and Disagree responses accepted. 

The following table shows the range of agreement of the 29 educa- 
tional psychology students on the 3 items. CIRCE’s Key was used 
for scoring 



Table 3 

Number of students passing the items 



No„ of 


Number 


Students 


of 


(out of 29) 


Items 


25-29/29 


5 


20-24/29 


13 


15-19/29 


9 




27 


10-14/29 


7 


5- 9/29 


4 




38 



On 27 of the 38 items, over fifty per cent of the graduate 
educational psychology students agreed with the key. This 
would seem rather high when one considers that the concepts of 
evaluation were relatively new to most of the student parti- 
cipants using the Urbana key , the range was from nineteen to 
twenty— eight passed with a mean of twenty— two— point— five (22.5) 
and a standard deviation of four-point-nine (4.9). Unfortunately, 
it is difficult to compare the results of the second edition 
with the first edition administered at Urbana, but the pre- 
attitudes of the graduate educational psychology students seem 
to be quite favorably disposed toward evaluation concepts, 
although this is only a guess since no comparative group exists. 
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Among the 29 "field-sample" studenst, fifteen 
volunteers completed the set of instruments 
and a calculation of their scores indicated 
the following correlation coefficients between 
the various revised instruments: 



Achievement Attitude 
Quiz Inventory 

Background Information .18 .10 

Achievement Quiz *34 



ACHIEVEMENT of PROGRAM'S GOALS 
Ac Affective Domain Objectives 

To elicit the CERLI staff's judgment concerning the achievement of 

the program's goals, a scheme encompassing the institute's curriculum, 

faculty and staff, and evaluation instruments was devised. (This 

plan was predicted on a schema developed by Chester S. Williams, 

"Preparing Teachers for the Disadvantaged: A Pre-Service Program 

Built on Hunches", Contemporar y Edu cation, 39:5, March, 1968, pp. 19 1-197. 

Because the summer institute's stated objectives (stimulus and per- 
formance) had not included the affective domain, the following were 
developed and superimposed: 

to develop an awareness of the important contribution 
evaluation makes to the educational endeavor 

to accept the evaluator role as one which assists 
educators rather than hinders them 

to want to assist in the further development of the 
field of educational evaluation 

to discuss evaluation and become more interested in 
the field 

to develop a climate among educators that is supportive 
of evaluation 



to assume a leadership role in the local educational 

setting 

to overcome any mental blocks against evaluation 

to enable the trainees to accept the ambivalent 
feelings encountered 

to view evaluation as a helping kind of relation- 
ship in the educational process 

to develop a professional relationship with evaluation 
agencies on a locals state and federal leve 3 and 

to try. 

Among the five CEKLI staff members who had attended and observed the 

Institute at Urbana, only two completed and returned the forms that 

they had been asked to fill out. (See Appendix: C- Curriculum 

Component.) The three who did not respond thought they could not 

validly judge the institute some six months later (March, 1969). 

Another staff member considered "ranking" a meaningless measure of 

curriculum and "positioning" the staff or faculty "impossible". 

B . CERLI Staff’s Evaluation . 

The fifth staff member (the Project Manager at CERLI and director 

of the summer institute) submitted the following critique of the 

post-institute evaluation instrument: 

Although this type of evlauation instrument would have some 
merit in the evaluation of another program 3 it is true that 
the length of time between the Institute and the administration 
of the forms to the staff was such that it would be difficult 
for one to complete . The format does show promise for future 

use 3 however . 

The project manager’s evaluation of components for achieving most 
of the objectives included: 

Curriculum: 



Stake’s Evaluation Model; Role Playing; 
Model Plans; Development of Model Plans 



Staff or Faculty: 



Douglas S jogren-Assistant Director, 

Human Factors Research Laboratory 
Colorado State University (Fort Collins) 



Robert Stake-Associate Director , 

CIRCE, University of Illinois 

Evaluation Instruments : 

Participant Critique Forms, Evaluation Plans, 
Attitude Survey, Opinionnaire 



APPENDIXES : 
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Appendix D: 
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Achievement Quiz 
Background Information Form 
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Evaluation Component 



APPENDIX A: 



ACHIEVEMENT QUIZ 



Directions: Please read the following research proposal and answer the 

fifty multiple-choice items that follow. These items are concerned for 
the most part with an evaluation of the proposal in terms of the pro- 
posal's objectives, outcomes, experimental design, instruments used, 
etc. Respond to the items as if you were in charge of the evaluation 
phase of the proposal and were responsible for giving some direction to 
the project director before the program commences. 



EVALUATION PLAN FOR INDEPENDENT STUDY 

Rationale: 

The academically talented child readily learns and retains information. 

By his very nature he constantly is seeking new challenges to his think- 
ing and creativity. Often, however, there is limited opportunity for him 
to apply the skills and information he has acquired in a way which is 
meaningful and rewarding to him. Thus, the need to establish an atmos- 
phere which allows the child the freedom of alternatives and the oppor- 
tunity for self-initiated activities becomes mandatory. With these facts 
in mind, the traditional concept of the school's role in directing the 
student's efforts must be modified. 

We are, therefore, faced with determining the kind of educational en- 
vironment that will allow our intellectually superior children to use 
their talents to the greatest efficiency so that they can grow up to be 
effective, contributing members of our society. 

Intended Antecedent: 

This project will involve an experimental and control group of intellec- 
tually superior sixth graders from the Junior High School. The students 
have been identified according to their scores on Large Thorndike 
Intelligence. 

The experimental and control groups will consist of ten students per 
group. They will be randomly selected from the top ten per cent of the 
gifted class as determined by their IQ score. The final determinant will 
be their expressed willingness to participate in the experimental study 
and parental support of the undertaking. 

Intended Transactions: 

This project will extend over a three year period beginning in September, 
1968 and terminating in June, 1971. 

The students selected for participation in the experimental group will be 
scheduled into accelerated classes as will those in the control group. 
Objective data, not included in the identification program, will be gathered 
at this time. In addition, data of a subjective nature will be collected. 
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Such things as attitude toward school teachers, classes, classmates and 
assignments will be noted along with noticeable strengths and weaknesses 
in self-initiative, self-discipline and independence. 

During these first weeks, the Project Director will be establishing the 
groundwork for a smooth transition from the existing program o e m 
program for both students and classroom teachers. Interested teachers, 

"the principal and the guidance counselor will be invited to serve with the 

Project Director in an advisory capacity. In addition, the Proj c 

will have contacted the parents of eligible students to explain the proposed 

program and enlist their support. 

After Involvement in assigned classes for approximately one month, the 
experimental group will meet with the Project Director and the Inte.dis- 

ciplinary Team who will provide an orientationtotheproposedprogramand 

indicate the range of options to which the student will be enti . Y 

effort will be made to encourage a trusting relationship between th 

students and the Team. The student association with these ****{ “resource 
should not be threatening. The students should look upon them as resour c 

people and guides in the process of educational decision-making. 

After the orientation session, the experimental group will meet with the 
team to consider the following: 



1, The role of the school 

2, Educational objectives 

3. Diagnosed strengths and weaknesses 

4. Areas of interest and needs 

At this point, each child in the experimental group will be given the option 
to determine: 



1. Which classes to attend 

2. What assignments to pursue 

3. How to use time away from class 

4. Where to spend time away from class 

5. With whom to spend time away from class 



The Project Director will be available to assist the students and consult 
with the students at all times. Members of the Team will also be avail- 

able upon request. 

As part of the developmental sequence of the program, the students’ initial 
options will be very limited. The progress of each child in adjusting to 
these will determine the point at which additional options providing for 
greater independence will be introduced. 



Intended Outcomes : 



This program has been designed to provide the School 
which will indicate the appropriateness of a program 
the gifted junior high population. 



System with data 
of this nature for 



We hope to show that: 

lo Given the freedom in which to operate, the intellectually 
superior child will eventually become a more productive 
individual, 

2, He will learn to make educational decisions and evaluate 
the consequences for these decisions in an environment 
where guidance is readily available. 

3, Involvement in this program will not adversely affect 
achievement; instead, the child will apply his skills 
and information in a way which is more meaningful and 
rewarding to him. 

4, Teachers will re-evaluate their role with greater em- 
phasis on the real needs of these students. 

Evaluation Procedure: 

RO X 0 
1 2 

RO 0 
1 2 

This study will consist of two groups of ten children who will be selected 
on a random basis from the top ten per cent of the sixth grade class, de- 
termined by their scores on the Lorge Thorndike Group Intelligence test, 

0X0 0X0 0X0 0X0 0X0 

1 2 3 4 5 

A pre and post test on the Iowa Test of Basic Skills will be administered 
at the beginning and end of the school year as a measure of achievement. 
Additional pre and post scores from other junior high gifted programs will 
be analyzed to see if this program has affected a change in achievement 
compared to other sixth grades in addition to determining any significant 
change between the two groups of ten in the experimental program. 

A measure of seif-direction will be developed to aid in identifying those 
children who display characteristics which seem to their ability to perform 
in an independent study program. If time permits, Suchman, Inquiry Train- 
ing and Critical Thinking will be administered during the year to determine 
whether a positive correlation between self-direction and these measures 
exists. It is hoped that by gathering as much data as possible, we will 
know more about this illusive quality of self-reliance, be better able to 
identify it and begin to know what will help children develop in this 
capacity. 

Cumulative record information will be gathered, conferences with parents, 
teachers and students will be annotated, class attendance records will be 
checked to determine how often children who are given options of whether 



or not to attend class actually option not to. 

An Independent Study questionnaire covering the major areas of Acceleration, 
Content, Resources and Evaluation plus a questionnaire on Change in Behavior 
and Relative Effects of Technique Compared to other Techniques including 
the areas of Learning— Content , Learning— Study Habits, Attitudes, and 
Social Interaction will be given to the group at the end of the school year 
to get an affective measure after one year of the program. 

A questionnaire to the community covering the full expanse of the educational 
programs will be developed to aid in understanding what the community views 
as important for their children and to determine how much understanding they 
do have in regard to the educational provisions of the district. 

The In-Service program for teachers will include a section devoted to 
evaluation as a "tool to understanding". They will be introduced to some 
of the basic skills of test construction and interpretation and will work 
on developing a rating scale or checklist for use in their classroom. 

The teachers will be introduced to self-assessment techniques such as 
Style of Teaching Inventory and Flanders Interaction Analysis. The 
Flanders will be used as a tool for noting changes in teacher-directed 
student-directed behavior as the school year progresses. Half-hour seg- 
ments will be collected once a week on a random basis. The school’s 
psychologist and the director of research and testing for the district 
will be an integral part of this section of In-Service. 



1. What important data is omitted from the proposal in regard to the 

control group? 

1. The selection procedure 

2. The activities they will pursue 

3. The identity of the control group 

4. The male-female composition of the control group 

2. The teachers that were assigned to the various groups would have what 
kind of impact on the learning outcomes? 

1. None at all 

2. Some, depending on the teachers’ orientation 

3. Much, especially if the teaching group is homogeneous 

4. Very much, especially if the teachers were randomly assigned 

3„ Assuming that the teachers were randomly assigned to the control and 
experimental groups, how could the director of the project best wipe 
out differneces? 

1. Matching intelligence 

2. Collecting data on teacher feelings toward the project 

3. In-service training 

4. Employing master teachers 
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4, What type of outcomes in the proposed study is most difficult to 
evaluate objectively? 

1, Study habits 

2, Attitudes 

3, Social interaction 

4, None of the above 

5, If a positive correlation between self direction and Inquiry Training 
and Critical Thinking exists in the proposed study, we could assume 

that 

1. Self direction caused the outcome 

2. Critical thinking is dependent on self direction 

3. Some relationship exists between the two 

4. A relationship exists, but caution is advised in interpretation 

6. In general, increasing the length of a test will make it more 

1. Valid 

2. Reliable 

3. Objective 

4. Diagnostic 

7. Which of the following data could not have been collected by interviews? 

1. Goal achievement 

2* Attitudes of the parents 

3, Goal orientation 

4. Perceptions of the program 

8 0 What statements can we safely make about the size of the samples in 

the various groups? . . ^ 

1„ Too small to make any sound conclusions regarding the outcomes 

2. Adequate if we are interested in verifying hunches 

3. Adequate if more control groups were included 

4. Adequate 

9. What is the most dynamic aspect of the program proposal? 

1. The background 

2. The educational process 

3. The outcomes 

4. The standards 

10. Would you say that the four program proposal objectives are stated as 



1 . 


Explicit 


2. 


Implicit 


3. 


Cognitive 


4. 


Psychomotor 


Do you 


consider the objectives 


1 . 


Global 


2 . 


Specific 


3c 


Behavioral 


4. 


Exp licit 



f 



12. Which of the following is least appropriate for educational evaluation? 

1. Educational process studies 

2. Proficiency measures 

3 . Attitude measures 

4. Cost studies 

13. An on-going evaluation of an existing program, formative evaluation, 
usually encompasses all but one of the following activities. Identify 
the activity that is usually not used for feedback and revision of a 
continuous program. 

1. Observation of student activities 

2. Pre and post testing 

3. Study of the logical consistency between goals, students, 
content, method and outcomes 

4. Objective experimental comparison among programs 

14. What do you consider to be a major weakness in the evaluation of the 
proposed study? 

1. No provision for continual assessment and goal or process 
revision 

2. The goals are not stated adequately 

3. Too many of the test instruments are of the home-made type 

4. The control groups are not adequately matched 

15. When would motivation be a significant variable that could affect the 
outcomes of the program proposal? 

1. When it is applied to the control groups only 

2. When it is applied to the experimental group only 

3. When it is applied to both groups 

4. When it is applied to neither group 

16. Why is the teachers’ attitude an important antecedent condition to 
consider? 

1. It could easily influence the results 

2. It would provide for greater goal achievement 

3* It will determine what teachers will stay with the program 
4. It is not important to consider 

17 „ In the proposal, what do the symbols RO^ x 0^ 

RO^ x 0_ mean? 

1. That the control and experimental groups are matched in all 
respects except for an intervening variable x 

2 . That both groups are quite dissimilar 

3. That the outcomes are expected to remain constant 

4. That a multiplier effect will operate with the experimental 
group 
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18. Would knowledge that they are participating in a research project have 
any effect on the students’ performance? 

1. Most definitely 

2. Probably 

3. Certainly not 

4. Don’t know 

19, Would knowledge of the experimental nature by the control group only 
affect the outcome of the program? 

1. Yes, but don't know to what extent 

2. Yes, and to a great extent 

3. No, since this variable has been controlled 

4. No, since the experimental design controls this 



20. Which aspect of the program proposal comes closest to being summative: 

i.e. evaluating the programs overall effectiveness? 

1. Matched groups 

2. Pre and post-test administration 

3. I.Q# test administration 

4. Renorts from the students 

21. Rational decisions about the program’s comparative worth can best be 
determined by which of the following activities? 

1. Comparing the program against a traditional program 

2. Comparing the program’s performance against a control group 
performance 

3. Comparing the performance of the experimental group with 
other schools 

4. Determining the cost of the program 

22. Would I.Q. scores of the participants in the experimental program 
have some effect on their achievement of independence and self 
direction? 

1. Yes. Since the program has picked the top sixth of the class, 
the I.Q. must be related to this characteristic. 

2. Yes, because studies indicate a high correlation between 
intelligence and the hoped-for achievement characteristics. 

3. No, because the standard error of measurement at the upper 
levels of I.Q. tests are usually greater than those at the 
other levels. 

4. No, because the group is too heterogeneous to tell. 



23. Which statement is most correct in regard to the size of the sample 
in the program proposal? 

1* The sample is too small for statistical inference 

2. The sample is adequate for statistical inference 

3. The t test would be an inadequate means of comparing the 
groups ’ performance 

4. Chi square would be a most adequate statistical test to use 
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24. What is the real danger in collecting judgments about programs? 

1. They are unreliable 

2. They are subjective 

3. They become the criterion for judging a program 

4. They are usually tested against an outside criterion 

25. Which of the following is the best criterion for judging the merits 
of a program? 

1. Standard 

2. Judgment 

3. Outcomes 

4. Cost 

26. What statement would best defend the usage of the Lorge-Thorndike 
intelligence test in the program proposal? 

1. The test had been standardized on an adequate sample 
population 

2. The test is highly reliable 

3. The test correlates quite high with a valid individual 
intelligence test 

4. The test is predictive of independent study success 

27. How would you describe the standards to judge the project proposal 
program’s outcomes? 

1. They are relative 

2. They are absolute 

3. They are non existent 

4. They are different for the experimental and control groups 

28. "As part of the developmental sequence of the program, the students' 
initial options will be very limited. The progress of each child 
in adjusting to these will determine the point at which additional 
options providing for greater independence will be introduced." 

How would you react to the above statement? 

1. Additional options will improve the design of the study 

2. Additional options will complicate the design and make control 
of the other variables quite difficult 

3. The amount of options can be correlated with the ten students’ 
success in the control group 

4. The amount of options can be correlated with the achievement of 
the ten students in the experimental group to determine the 
causal impact of the options on achievement 

29. Good Housekeeping Seal of Approval is to new products as field testing 

is to . 

Matched design 
Instrument development 
Variable selection 
Group experiment 
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For what phase of the project proposal is it most important to have 
norm data? 

1. Rating scale development 

2. Checklist development 

3. Intelligence test selection 

4. Self assessment techniques selection 

31. One of the intended outcomes of the program is stated as follows: 

Given the freedom in which to operate, the intellectually superior 
child will eventually become a more productive individual. How would 
you best measure the attainment of this outcome? 

1. By administering an independent study questionnaire 

2. By administering the Iowa Test of Basic Skills 

3. By providing appropriate experiences 

4. By observing youngsters 

32. If the evaluator had to judge the contingency between the educational 
experiences and the outcome, what would be his best point of reference? 

1. Buros Mental Measurement Yearbook 

2 . Encyclopedia of Educational Research 

3- Taxonomy of Educational Objectives 

4. Psychology Today 

ITEMS 33-36 ARE TO BE ANSWERED TRUE OR FALSE— 1 for true, 2 for false. 

33. Educational evaluation is essentially the same as educational 
research in terms of techniques used and in terms of questions to be 
answered. 

34. Questionnaire information is the least reliable and useful information 
evaluators collect. 

35 ., The size of the samples, method of drawing it, and other features of 
the survey design will not be affected by the kind of analysis to be 
made of the results. 

36. Archives might include examining science-teacher-of-the year candidates’ 
careers . 

37 . Evaluative Criteria is to accreditation study as Campbell and Stanley’s 

handbook is to ,• 

1. Rating scales 

2. Statistics 

3. Questionnaires 

4. Experimental research 

38. The Chi square technique is commonly used for 

1. Describing groups in terms of fine measurement data 

2. Testing hypotheses regarding "fine measurement" data 

3. Describing groups in terms of frequency counts 

4. Testing hypotheses regarding frequency counts 
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39. The Q Technique and conventional factor analysis are both techniques 
for 

1. Analyzing profiles of students 

2. Clustering "like things" together 

3. Comparing large numbers of groups 

4. Evaluating instructional television 

40. The process of generalizing from sample data to population conditions 
while at the same time specifying the investigators’ confidence in 
drawing correct conclusions is known as 

1. Summative evaluation 

2. Interaction analyses 

3. Statistical inference 

4. Taking a calculated risk 

41. Which of the following is usually not considered a major area of 
specialization for the educational research methodologist? 

1. Measurement, testing, instrumentation 

2. Research design, experimental controls 

3. Statistical description and inference 

4. Cost-benefit analysis, program evaluation 

42. Test techniques are generally preferred to observational techniques, 
when both are available for the testing purposes, because the former 
are 

1. More apt to yield measures 

2. Perceived as a test by the student, thus more apt to be based 
on a motivated performance 

3. Applicable to a wider variety of personal traits 

4. More apt to yield reliable scores 

43. Which of the following scores appearing in a student’s record would 
be most meaningful without further reference to the group? 

1. 23 items correct in an English test of 40 items 

2. 30 items wrong in an algebra test of 50 items 

3. 100 words per minute in a typwriting test 

4. Omitted ten items in each of the English and algebra tests 

44. Problems arise in attempting to develop measures of ultimate goals 
mainly because 

1. Measurement methods have not given proper weight to all goals 

2. Teachers have been reluctant to depart from traditional testing 
methods 

3. Group norms with which to compare results are not available 

4. Such goals concern behavior not usually observable under class- 
room conditions 
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order to compute a correlation coefficient between traits A and B, 
is necessary to have 

1. Measures of trait A on the group of persons, and of trait B 
on another 

2„ One group of persons, some who have both A and B, some with 
neither and some with one but not the other 

3. Two groups of persons, one which could be classified as A or 
not A, the other as B or not B 

4. Measures of traits A and B on each person in one group 



46. Test norms are most satisfactory when the sample of pupils or students 
used in establishing the norms 

1. Consists of nearly all pupils or students taking the test 
prior to the time the norms are published 

2. Is representative of a clearly defined population with which 
it is appropriate to make comparisons 

3. Ranges over all the grade levels in which the test is likely 
to be used 

4. Includes all schools volunteering to participate in the 
standardization testing 



47. Outcomes include which of the following activities? 

1. In-service training of teachers 

2. Demonstration of not planned-for behavior 

3. Administration of a pre-test 

4. Preparation of goals 



48. Is it possible to measure and observe the outcomes of the program 
proposal immediately after the school term is completed? 

1. Yes, because a delay will cause a reversal of behavior 
2* No, only a longitudional study can do this 
3* Yes, to a certain extent 
4. None of the above is correct 

49. Why must unintended program outcomes also be carefully considered? 

1. They would illustrate the impact of the program 
2* They help define the nature of the program 
3» They should not be considered at all 

4. They could negate the effects of the desirable outcomes 

50. What is the effect of the parents on selecting the options and 
determining the behavior of the program proposal participants? 

1. It is a variable to consider 

2. It would cause little effect 

3. Too many major variables have already been identified 

4. Parents would have little impact here 
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APPENDIX B: BACKGROUND INFORMATION FORM 

The evaluation institute you will be attending in the near future is 
designed to help each of the participants achieve a high degree of mastery 
in the evaluation area. To accomplish this we must first have some informa- 
tion on your background so that the institute can be planned accordingly. 
Therefore, will you complete the following questionnaire and teturn it 
by the . 

1. Nam e 

2 . What is your present job title? 

3. How many years have you had this particular job? 

4. How many years have you worked in the field of education? 

5. If you are not a teacher, please describe your job. 



6. Highest educational degree received? 

7. What was your major area of study in your graduate program? 



8. What has been your exposure to statistics at the college level? 

A. None 

B. Part of a course — i.e. educational psychology 

What course or courses? 

C. One full course 

D. More than one course 

How many? 

9. When were the statistics courses taken? 

A. One year ago 

B. Two years ago 

C. Three years ago 

D. More than three years ago 

10. How many tests and measurement courses have you taken? 



A. 


None 




B o 


Part of a 


course 


C. 


One full i 


course 


D, 


More than 


one course 




How many? 
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11 • Have you ever established goals for a program or worked on a committee 
doing so? 

A. Yes 

B. No 

C . Don T t remember 

12. Describe your experience with any phase of evaluation? _____ 



13. When sampling behavior, do you feel more comfortable with 

A. Tests 

B. Observations 



14. Do you regularly prepare objectives and statements of expected outcomes 
for your courses? 

A. Yes 

B. No 



15. Do you feel the outcomes expected are more important than a consideration 
of the "raw material" going into the program — the students’ background? 

A. Yes 

B. No 

C . Don T t know 



16. If you were asked to conduct an evaluation of a program in your school, 



how would proceed? 



17. What other courses have you had that would help prepare you for eval 
uation? 



18 , Whom might you ask to assist you in the evaluation of an educational pro 
gram? 



19. Please list any books or articles (and authors, if possible) that you may 
have read in any of these areas: 

Tests and Measurement 



Educational Psychology 
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Statistics 



Instructional Objectives 



20. Following is a list of terms and phrases which are important to 
one undertaking a study of the role of the evaluator. Indicate 
your degree of understanding of each according to the following 
code; 

CU Completely understand the item to a degree where 
it can be applied 

SU Some understanding of the item 

VU Vague understanding of the item 

NU No understanding of the item 



1 . 


Achievement test 


15. 


Diagnostic test 


2. 


Alternate-form reliability 


16. 


Difficulty value 


3. 


Aptitude 


17. 


Discriminating power 


4. 


Arithmetic mean 


18. 


Distribution 


5. 


Average 


19. 


Enabling objective 


6. 


Battery 


20. 


Equal interval scale 


7. 


Behavioral change 


21. 


Equivalent form 


8. 


Ceiling 


22. 


Face validity 


9. 


Class interval 


23. 


Factor analysis 


10. 


Coefficient of correlation 


24. 


Forced-choice item 


11. 


Correlation 


25. 


Formation evaluation 


12. 


Criterion 


26. 


Grade norm 


13. 


Cross-validation 


27. 


Group test 


14. 


Deviation 


28. 


Individual test 



29. 


Input 


55. 


Rationale of the program 


30. 


Intelligence quotient 


56 . 


Raw score 


31. 


Inventory test 


57. 


Recall item 


32. 


Item 


58. 


Reference population 


33. 


Item analysis 


59. 


Reliability 


34. 


Median 


60. 


Reliability coefficient 


35. 


Mental age 


61 . 


Representative sample 


36. 


Mode 


62. 


Scale 


37. 


Normal distribution 


63. 


Skewness 


38. 


Norms 


64. 


Speed test 


39. 


Objective test 


65 . 


Split-half coefficient 


40. 


Objectivi ty 


66 . 


Standard deviation 


41. 


Percentile rank 


67. 


Standard error 


42. 


Percentile 


68. 


Standard error of 


43. 


Performance test 


69. 


measurement 
Standard score 


44. 


Personality test 


70. 


Standardized test 


45. 


Power test 


71. 


Stanine 


46. 


Practice test 


72. 


Stratified sample 


47. 


Profile 


73. 


Summative evaluation 


48. 


Prognostic test 


74. 


t-test 


49. 


Projective technique 


75. 


Test-retest coefficient 


50. 


Quartile 


76. 


Transmutation of scores 


51. 


Random sample 


77. 


True-false item 


52. 


Range 


78. 


True score 


53. 


Rank order scale 


79. 


Validity 


54. 


Rating Scales 


80. 


Variability 
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APPENDIX C 



Cupriculicm Com2£neirbs 



Dear Staff Member: 

I am attempting to discover the relationship between 
the curriculum conponents of last summer’s evaluation institute 
held at Urbana and the planned-for outcomes. 

Please read each objective and select the curriculum 
component that most helped meet that objective. The number 
of that component should be inserted in the box under t e 
heading First. Then select the next most useful component 
and insert the number of the curriculum component under the 
Second heading. Finally, select the third curriculum com- 
ponent and insert it under the appropriate heading. 

Proceed to the next objective. Thank you. 



CURRICULUM COMPONENTS 



1 . 


Evaluation model 


15. 


2. 


Videotapes 


16. 


3. 


Individual work session 


4. 


Group work session 


17. 


5. 


Observation and testing 


6 . 


Resource material 


18. 


7. 


Classroom observation 


19. 


8. 


Textbook evaluation 


20. 


9. 


Research design 


21. 


10. 


Unobtrusive measures 


22. 


11. 


Statistical problems 


12. 


Role playing 


23. 


13. 


Judgments 




14. 


Panel on evaluation 
problems 





Measuring cognitive 
outcomes 

Questionnaire con- 
struction 
Survey procedures 
Test Development 
Interview training 
Scaling 

Evaluation plan 
critique 
Model plans 
Develop model plan 



CURRICULUM COMPONENTS 
Order of Importance 



1 . 



AFFECTIVE DOMAIN 

. 1 To develop an awareness of the im- 
portant contribution evaluation 
makes to the educational endeavor 

.2 To accept the evaluator role as one 
which assists educators rather 
than hinders them. 

.3 To want to assist in the further de- 
velopment of the field of educa- 



First Second Third 




tional evaluation. 



CURRICULUM COMPONENTS 
Order of Importance 



First Second Third 

AFFECTIVE DOMAIN (continued) 

.4 To discuss evaluation and become more 
interested in the field. 

,5 To develop a climate among educators 
that is supportive of evaluation. 

.6 To assume a leadership role in the 
local educational setting. 

.7 To overcome any mental blocks 
against evaluation. 

.8 To enable the trainees to accept the 
ambivalent feeling. 

.9 To view evaluation as a helping kind 
of relationship in the educational 
process . 

.10 To develop a professional relationship 
with evaluation agencies on a local, 
state and federal level. 

.11 To try. 




STIMULUS DOMAIN 



.1 



. 2 



.3 



»4 



.5 



To encounter the logical arguments for 
gathering evaluation information to 
assist in the making of rational 
decisions in the local project. 

To encounter an evaluation model 
around which the local evaluation 
efforts could be organized. 

To encounter a discussion of evalua- 
tion problems featuring the view- 
points of experienced evaluators , 
researchers, and administrators. 

To encounter a growing opportunity for 
sharing local project findings with 
staff members of other projects. 

To encounter direction to an abundance 
of resources, personal and otherwise, 
to aid in the conduct of evaluation 
at the local level. 



1 














1 














4 
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PERFORMANCE DOMAIN 

.1 To design components of an evaluation 
plan for local use. 

.2 To make a table of contents for a sum- 
mative evaluation report of his own 
local gifted program. 





CURRICULUM COMPONENTS 
Order of Importance 





First 


Second 


Third 


PERFORMANCE DOMAIN (continued) 




.3 To improve his facility for using the 












language and concepts of measure- 












ment and evaluation. 


23 










.4 To prepare to try out certain. stand- 




ardized classroom observation 












techniques that can be useful in 


5 










evaluation studies. 












.5 To examine prototype evaluation re- 




ports and read selections from 












the literature on evaluation. 












.6 To work out solutions to a series of 




problems designed to stimulate the 












conditions and circumstances of 


22 




23 






local gifted projects in Illinois # 













APPENDIX D 



Hr 



Staff op Faculty Component 



Dear Staff Member: 



This time I am going to present you with the same 
list of objectives as before, but instead of asking you to 
give the curriculum components that were of most importance 
in achieveing the objective, I would like you to select the 
three individuals, in order of importance, who contributed 
significantly to the attainment of the objective. Thanks 
again. 



STAFF OR FACULTY COMPONENT 



1. 


R. Cunningham 


7. 


T, Kerins 


2. 


T. Denny 


8. 


S. Lap an 


3. 


S. Feldman 


9. 


R. Stake 


4. 


S. Gotch 


10. 


D. Sjogren 


5. 


T. Hastings 


11. 


G. Summers 


6. 


E. House 


12. 


B. Wilson 



STAFF OR FACULTY COMPONENT 
Order of Importance 



1. AFFECTIVE DOMAIN 

.1 To develop an awareness of the im- 
portant contribution evaluation 
makes to the educational endeavor. 

.2 To accept the evaluator role as one 
which assists educators rather 
than hinders them. 

.3 To want to assist in the further de- 
velopment of the field of educa- 
tional evaluation. 

.4 To discuss evaluation and become 
more interested in the field. 

.5 To develop a climate among educators 
that is supportive of evaluation 

.6 To assume a leadership role in the 
local educational setting. 

.7 To overcome any mental blocks 
against evaluation. 

.8 To enable the trainees to accept the 
ambivalent feeling . 

.9 To view evaluation as a helping kind 
of relationship in - he educational 
process 



First Second Third 
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STAFF OR FACULTY COMPONENT 
Order of Importance 










! 



AFFECTIVE DOMAIN (continued) 

.10 To develop a professional relationship 
with evaluation agencies on a local, 
state and federal level 
.11 To try. 

2. STIMULUS DOMAIN 

.1 To encounter the logical arguments for 
gathering evaluation information to 
assist in the making of rational 
decisions in the local project. 

.2 To encounter an evaluation model 
around which the local evaluation 
efforts could be organized. 

.3 To encounter a discussion of evalua- 
tion problems featuring the view- 
points of experienced evaluators, 
researchers, and administrators. 

.4 To encounter a growing opportunity for 
sharing local project findings with 
staff members of other projects. 

.5 To encounter direction to an abundance 
of resources , personal and otherwise, 
to aid in the conduct of evaluation 
at the local level 



First 


Second 


Third 
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10 




2 








9 




10 
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9 




10 
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10 


1 


l 2 



3. PERFORMANCE DOMAIN 

.1 To design components of an evaluation 

plan for local use. 

.2 To make a table of contents for a sum- 
mative evaluation report of his own 
local gifted program. 

.3 To improve his facility for using the 
language and concepts of measure- 
ment and evaluation. 

.4 To prepare to try out certain stand- 
ardized classroom observation 
techniques that can be useful in 
evaluation studies. 

.5 To examine prototype evaluation re- 
ports and read selections from 
the literature on evaluation. 

.6 To work out solutions to a series of 
problems designed to stimulate the 
conditions and circumstances of 
local gifted projects in Illinois. 
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APPENDIX E 



Evaluation Corr^oneYit 



Dear Staff Member: 

I am going to ask you to make one last examination 
of the objectives that were supported during the evaluation 
institute at Urbana. This time I would like to introduce 
the evaluation component and to have you determine which of 
the nine evaluation components listed were used to evaluate 
the objectives and to what extent those used did so. Use 
the following key and assign a letter to each objective 
below each of the evaluation components. 

Key to be used: 

E - Excellent F - Fair 

VG - Very Good P - Poor 

G “ Good NA - Not Applicable 

Example: if the participant critique form did an excellent 
job of evaluating objective 1.1, place an IS in the ce If 
the Observer's report was only fair in evaluating the same 
objective, place an F. in the appropriate cell. Thank you. 



EVALUATION COMPONENT 



1. Self Perception 

.1 Participant Critique Form 

2. Staff Evaluation 

.1 Observer’s Report 
.2 Evaluation Plan 

3. Peer Evaluation 

.1 Participant Critique Form 
.2 Participant Interview Schedule 
.3 Role Playing 

4. Comparative Evaluation 

. 1 Achievement 
.2 Attitude 
. 3 Opinionnaire 
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